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PURPOSE 


The Office of the Inspector General (OIG), in accordance with the Transition Agreement and 
pursuant to its Audit and Review Plan, has completed its Training Evaluation and Management 
System IT (TEAMS II) Audit (Audit). A mandate of the Transition Agreement is to review the 
processing of Action Items for compliance with Special Order No. 28, 2008, entitled “Duty to 
Conduct and Document Individual Performance Assessments - Revised,” dated August 29, 2008. 
This Special Order describes how TEAMS II is to be used as a risk management system, 
primarily through the processing of Action Items. Hence, the purpose of this Audit was to 
determine if these Action Items were appropriately and timely processed. 


BACKGROUND 


The Transition Agreement requires a review of the TEAMS II system to determine if the 
Department is utilizing the system “in the manner in which it was intended — an early warning or 
risk management system.”' In part, the Transition Agreement requires a review of Threshold- 
Activated Action Items “to verify that supervisors are conducting a review of TEAMS II 
information to detect any pattern or series of incidents that indicate that an officer may be 
engaging in at-risk behavior.” 


TEAMS II is a system that provides a summary of an officer’s work performance history and 
allows supervisors to compare an officer’s work performance history to other officers who 
perform similar work. TEAMS II uses several sources of information to track and compare the 
performance of officers; some of those sources track crimes and arrests, claims and lawsuits, 
commendations, collisions, traffic citations, training, vehicle pursuits, pedestrian and vehicle 
stops, complaints, use of force incidents and risk management information. This information is 
tracked by TEAMS II and appears on an officer’s TEAMS II report. 


To identify officers who may be at risk of committing misconduct, supervisors are required to 
evaluate TEAMS II reports for officers at four determined junctures: each time an officer 
transfers to a new assignment, receives an annual performance evaluation, reaches a higher 
activity level than his/her peers, or when the supervisor chooses to conduct a performance 
assessment. The evaluation includes an analysis of all relevant information to detect any pattern 
or series of incidents that may indicate that an officer may be engaging in at-risk behavior. 


The method developed to track a supervisor’s review and evaluation of TEAMS II reports is the 
Action Item. The four types of System-Generated Action Items reviewed in the Audit were 
Transfer Action Item, Performance Evaluation Report Action Item, Threshold-Activated Action 
Item and Supervisor Action Item. 


The Audit tested each of the four types of System-Generated Action Items in order to assess 
whether supervisors used the information contained in TEAMS II reports to analyze potential 





' Transition Agreement, Page 8, Line 16. 


* Id., Page 8, Line 25. 
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patterns and practices of behavior that may lead to officer misconduct as required by the 
protocols established in Special Order No. 28, 2008. 


METHODOLOGY AND SCOPE 


The OIG conducted this Audit in accordance with generally accepted government auditing 
standards. These standards require that the Audit is adequately planned, performed and 
supervised, and that sufficient, appropriate evidence is obtained by applying review procedures 
which assess whether a reasonable basis for the findings and conclusion were contained herein. 


The OIG used Special Order No. 28, 2008, as the standard for testing the relevant samples. 
Details for each objective’s methodology, sample and scope are provided in the Detailed 
Methodology section of each objective. The following chart depicts a summary of results. 





ACTION ITEM CATEGORY AND OBJECTIVE COMPLIANCE RATE 





A. Threshold-Activated Action Items (AIs) 
Objective Al: Determine if Als were adequately justified, reviewed and 





documented in accordance with Special Order No. 28, 2008. ee’) 
Objective A2: Determine if Als with corrective action disposition(s) were 
adequately explained and justified in accordance with Special Order No. 28, 39% (14/36) 


2008. Also determine if these corrective action disposition(s) represented 
new action taken as a result of the AI review. 





Objective A3: Determine if Als were completed within 60 days of creation. 83% (454/550) 





B. Transfer Action Items (TAIs) 





Obj. B1: Determine if the TAI and related TEAMS Evaluation Report (TER) 


forms were completed in accordance with Special Order No. 23, 2003. ee) 





Obj. B2: Determine if the TAIs and TER forms were completed within 10 


days of publication of the related Transfer Order. AA ARN] 





Obj. B3: Determine if TAIs were generated for all officer transfers. 100% (90/90) 





C. Performance Evaluation Report Action Items (PERs) 





Obj. C1: Determine if the PERs and related Standards Based Assessment 
forms evaluated the supervisor’s performance in accordance with Department 100% (21/21) 
Manual Section 3/760.20. 





Obj. C2: Determine if Performance Evaluation Report Action Items were 


100% (40/4 
generated for related annual performance evaluations. eg ue 





Obj. C3: Determine if PERs were completed within 60 days. 69% (1,526/2,201) 





D. Supervisor Action Items (SAIs) 





Obj. D1: Determine if SAI dispositions were adequately reviewed and 


justified in accordance with Special Order No. 28, 2008. eens) 








Obj. D2: Determine if there was evidence that corrective action was taken for 


0 
: : : A an 1 
SAIs with corrective action dispositions. AWRY) 
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DETAILED RESULTS 


A. Threshold-Activated Action Items (Action Items) 


Objective Al: Determine if Action Items were adequately justified, reviewed and documented 
in accordance with Special Order No. 28, 2008. 


Background 


The TEAMS II system used to conduct performance analysis is the Risk Management 
Information System (RMIS). RMIS assesses a variety of areas, including an officer’s uses of 
force, complaints, claims/lawsuits, preventable traffic collisions and vehicle pursuits. Each area 
is numerically measured by being compared to stops and arrests per officer known as thresholds. 
The thresholds are then averaged by peer groups, comprised of all officers who perform similar 
job functions. RMIS automatically notifies the officer’s assigned supervisor via an Action Item 
if the officer exceeds the average for any threshold, thereby requiring the supervisor to review 
and analyze the officer’s TEAMS II report. 


Special Order No. 28, 2008, outlines the requirements for completion of an Action Item. 
Supervisors must briefly document all Action Item areas and are required to conduct an analysis 
of the officer’s entire work history to determine if a pattern of conduct or at-risk behavior may 
exist. Neither the criterion for a pattern of conduct nor the definition of at-risk behavior is 
defined by Special Order No. 28, 2008. In addition, the supervisor is required to document a 
comparison of the officer’s work performance against his/her peer group. In order to assist with 
that analysis, TEAMS II automatically links the Summary of Employee Activity report to the 
Action Item. Based on the analysis, the supervisor is then required to determine an appropriate 
course of action for the officer and provide a written justification for the disposition.’ Finally, 
the supervisor must have a discussion with the affected officer regarding the disposition and 
document the discussion in the Action Item narrative. 


To promote consistency in documenting reviews of officers work performance, TEAMS II 
automatically includes the following eight information prompts to address the aforementioned 
requirements in the Action Item narrative: 


(1) Provide a brief summary of each use of force within the evaluation period. 

(2) Provide a brief summary of each complaint within the evaluation period. 

(3) Provide a brief summary of each vehicle pursuit and preventable traffic collision within 
the evaluation period. 

(4) Provide a brief summary of each claim/lawsuit within the evaluation period. 

(5) Compare the officer’s performance against that of similar officers. 

(6) Conduct an analysis of all the events as a whole and determine if there is a “pattern of 
conduct.” 

(7) Justify the disposition selected. 





> A supervisor may choose one or more from the following dispositions: no action, commendation, informal 
meeting, training, special evaluation reports, modified field duties, assigned to non-field duties, Risk Management 
Executive Committee referral, directed Behavioral Science Services referral, comment card, notice to correct 
deficiencies, and complaint. 
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(8) Provide a brief summary of the discussion with the affected officer regarding the 
supervisor’s review and selection of disposition. 


Detailed Methodology 


To select a current and thereby relevant sample of Action Items, the OIG obtained a report from 
TEAMS II RMIS staff of all Action Items completed from January 1, 2009, through September 
8, 2009. The total population consisted of 1,861 Action Items. The OIG then deselected all 
corrective action dispositions, as those dispositions were tested in a separate sample for 
Objective A2.* The OIG then judgmentally selected the period from June 1, 2009, to September 
8, 2009, for this objective. 


The selected time period contained 653 Action Items. From the 653 Action Items, the OIG 
selected a random sample of 40 Action Items by using a 95% one-tail confidence level, with a 
6% expected error rate, and a 6% plus-precision. 


Three RMIS reports were reviewed for each Action Item: the Action Item History report, 
TEAMS II report and Summary of Employee Activity report.” Each Action Item History report 
contains the documented narrative and comments of the reviewers. Each Action Item was 
examined to determine if the supervisor and reviewers had adequately documented their review 
in accordance with Special Order No. 28, 2008. Further, each Action Item was compared to the 
TEAMS II report and Summary of Employee Activity report in order to verify the narrative and 
comments. 


The OIG noted that Special Order No. 28, 2008, does not require a specific due date for an 
Action Item review, but the Order does indicate that “the Risk Management Information System 
will automatically assign a due date to the AI.” TEAMS II RMIS staff assigns a due date 60 
days from the creation of the Action Item. The Action Item investigation should be conducted 
and reviewed within that time period. 


The following tests were used to review the sample Action Items: 


Test 1: Determine if there was a brief summary of each use of force, complaint, claim/lawsuit, 
vehicle pursuit, and traffic collision occurring within the evaluation period. 


Test 2: Determine if there was an analysis of the events as a whole used to determine if there is a 
pattern of conduct. 


Test 3: Determine if there was a documented comparison of the officer's performance against 
officers in the same peer group, organization or similar job assignment and explain any 
significant differences between the affected officer's performance and that of similar officers. 





4 Corrective action dispositions include training, modified field duties, assigned to non-field duties, Risk 
Management Executive Committee referral, directed Behavioral Science Services referral, notice to correct 
deficiencies, special evaluation reports and complaint. 


5 An Action Item History report is the documentation of an Action Item investigation. 
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Test 4: Determine if there was justification for the disposition selected. 


Test 5: Determine if there is a brief summary of the discussion with the affected officer 
regarding the supervisor's review and disposition selection. 


Testing Results 


The Department was 93% compliant with this objective. Thirty-seven of the 40 Action Items 
tested contained the required summary. All three exceptions were found as a result of Test 1; 
each of those three Action Item narratives did not contain the required event information which 
was available in the affected officer’s TEAMS II report. Details for those exceptions are as 
follows: 


The supervisor completing the first Action Item addressed the event that initially triggered the 
Action Item but did not discuss the officer’s complaints, uses of force, pursuits, or 
claims/lawsuits.° The supervisor completing the second Action Item did not document two prior 
use of force incidents.’ The supervisor completing the third Action Item incorrectly reported that 
no preventable traffic collisions occurred during the evaluation period. ê 


Objective A2: Determine if Action Items with corrective action disposition(s) were adequately 
explained and justified in accordance with Special Order No. 28, 2008. Also determine if 
these corrective action disposition(s) represented new action taken as a result of the Action 
Item review. 


Background 


According to Special Order No. 28, 2008, supervisors must provide justification for the 
disposition selected for an Action Item. A supervisor may choose one or more of the following 
dispositions: no action, commendation, informal meeting, training, special evaluation reports, 
modified field duties, assigned to non-field duties, risk management executive committee 
referral, directed Behavioral Science Services referral, comment card, notice to correct 
deficiencies, and complaint. 


Special Order No. 28, 2008, uses the following definitions for the reason each disposition should 
be selected: 


No Action - used when no pattern of behavior posing potential risk was identified in the Action 
Item investigation or when corrective action was taken prior to the Action Item and no new 
action is required. 





é AI No. 528393 (Complaint Form Nos. 09-002430, 09-002345 and UOF Case Nos. 2008378, 2008341, 2008203) 
7 AI No. 527853 (Use of Force Case Nos. 2008279, 2008839) 


8 AI No. 526552 (Traffic Collision No. 081037373) 
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Commendation - used when the Action Item investigation reveals that the work history of an 
officer is above average and should be commended. 


Informal Meeting- used when a supervisor meets with the officer and conducts an informal 
counseling session that does not result in any further action as a result of the Action Item 
investigation. 


Training - used when the Action Item investigation identified a need for formal training. 


Special Evaluation Reports - used when the Action Item investigation determined an officer 
needs further monitoring. 


Modified Field Duties - used when the Action Item investigation identifies a need for 
modification within an officer’s work duties. 


Assigned to Non-Field Duties - used when the Action Item investigation determined that an 
officer should be removed from the field and placed in a non-field duty assignment. 


Risk Management Executive Committee (RMEC) Referral - used when the Action Item 
investigation revealed that an officer should be referred to RMEC for special monitoring.’ 


Directed Behavioral Science Services (BSS) Referral - used when the Action Item investigation 
revealed that an officer should be referred to BSS for mental health services. 


Comment Card - used when the Action Item investigation requires that a supervisor complete a 
comment card. ° 


Notice to Correct Deficiencies (NTCD) - used when the Action Item investigation determines 
that a NTCD is necessary."! 


Complaint - used when misconduct is identified through the Action Item investigation and a 
personnel complaint should be initiated due to the newly discovered misconduct. 





? RMEC reviews officer performance to assess liability and also directs non-punitive corrective actions to address 
performance and/or behavior. RMEC also makes recommendations to the Chief of Police to issue, renew, or rescind 
Long or Short-term duty (L/STDR) restriction letters. Both letters restrict an officer to non-field duties with no 
public contact, arrests, or court testimony. LTDR letters are reviewed annually and STDR are reviewed semi 
annually. 


10 Comment cards are used to document one of the following: an incident or behavior that was notably good or bad; 
a procedure, policy, or expectation was explained; specific training presented for a particular purpose; or that a 
particular act which was against policy and/or procedure was unacceptable. 


'' A NTCD is used to warn or admonish an officer of deficient performance or that a particular act was a violation of 
a tule, policy, procedure, or expectation, and that continuing the deficient performance or repeating the particular act 
(or acts like it) will result in at least a set of particular adverse consequences. 
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Additionally, Special Order No. 28, 2008, specifically states that the disposition of an Action 
Item should reflect a new action being taken as a result of the Action Item investigation and 
should be supported by the Action Item narrative. Further, any action taken prior to the creation 
of the Action Item for the investigated event(s), not requiring further action, should be closed 
using the disposition of no action. 


The OIG determined the corrective action dispositions employed in the sample of Action Items 
reviewed to be: assigned to non-field duties, directed BSS referral, modified field duties, special 
evaluation reports, training, complaint, RMEC referral, and NTCD. Based on definition, the 
OIG further determined that there were corrective actions which should be considered a higher 
risk. The higher risk corrective action dispositions included complaint, RMEC referral and 
NTCD. 


Detailed Methodology 


The OIG obtained a report from TEAMS II RMIS staff of all Action Items completed from 
January 1, 2009, through September 8, 2009. The total population consisted of 1,861 Action 
Items. The OIG sorted the listing by officer serial number and disposition type. The OIG then 
eliminated all non-corrective dispositions which were previously tested in objective A1. The 
OIG further separated all Action Items with the dispositions of complaint, RMEC referral, and 
NTCD in order to test all high risk corrective action dispositions separately for this objective. 


The OIG used the entire population of 56 Action Items with corrective action dispositions to 
obtain a random sample of 30 Action Items using a 95% one-tail confidence level, with a 6% 
expected error rate, and a 6% plus-precision. The OIG also identified six Action Items with high 
risk corrective action dispositions. Thus, the OIG reviewed a total of 36 Action Items for this 
objective. 


Three RMIS reports were reviewed for each Action Item: the Action Item History report, the 
TEAMS II report and the Summary of Employee Activity report. Each Action Item History 
report contains the documented narrative and comments of the reviewers. There are several 
levels of review for each Action Item investigation, the employee’s supervisor(s), the 
commanding officer and the commander/civilian equivalent. Therefore, each report was 
examined to determine if the supervisor and reviewers had adequately documented their review 
in accordance with Special Order No. 28, 2008. Further, each Action Item was compared to the 
TEAMS II report and Summary of Employee Activity report in order to verify the narrative and 
comments. 


The following tests were used to review the sample Action Items: 


Test 1: Determine if the corrective action disposition was adequately explained and justified in 
the Action Item narrative in accordance with Special Order No. 28, 2008. Additionally, 





'? Non-corrective action dispositions include: no action, commendation, and informal meeting. 


13 An Action Item History report is the documentation of an Action Item investigation. 


Training Evaluation and Management System (TEAMS) II Audit 
Page 8 of 19 
1.0 


determine if the disposition adequately supported the information in the Summary of Employee 
Activity and TEAMS II reports. 


Test 2: Determine if corrective action dispositions represented new action taken as a result of the 
Action Item investigation. 


Testing Results 


The Department was 39% compliant with this objective. Fourteen of the 36 Action Items tested 
contained the required information. All 22 exceptions were found as a result of Test 2. Each of 
those Action Item narratives did not represent a new action taken as a result of the Action Item 
tested. Details for those exceptions are as follows: 


Nine Action Items with the disposition of “assigned to non-field duties” referenced actions that 
had been imposed prior to the generation of the Action Item, rather than taken as a result of the 
Action Item investigation.'* As the Action Items did not require any new action, the proper 
disposition was “no action.” 


Four Action Items with the disposition of “complaint” referenced the complaint which generated 
the Action Item." According to Special Order No. 28, 2008, the complaint disposition is used in 
the event new misconduct is identified during review of the Action Item and a personnel 
complaint is required. As the Action Items did not require any new action, the proper disposition 
was “no action.” 


Five Action Items had the disposition of “training” assigned as a result of categorical use of force 
investigations.'® This action was not the result of a new action, but rather, the result of a finding 
made by the Board of Police Commissioners. As the Action Items did not require any new 
action, the proper disposition was “no action.” 


Three Action Items had the disposition of “training” for an officer who had already received 
training in a prior Action Item." As the Action Items did not require any new action, the proper 
disposition was “no action.” 

Objective A3: Determine if Action Items were completed within 60 days of creation. 


Background 


The OIG noted that Special Order No. 28, 2008, does not require a specific due date for an 
Action Item investigation. The Order only indicates that “the Risk Management Information 





* AI Nos. 520704, 523521, 514074, 515950, 521248,513977, 512556, 516732, 522872 
5 AI Nos. 517482, 511471, 528416, 515502 


é AI Nos. 517023, 526495, 526490, 526493, 529589 











7 AI Nos. 524392, 528636, 520807 
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System will automatically assign a due date to the AI.” TEAMS II staff advised the OIG that 
RMIS assigns a due date 60 days from the creation of the Action Item. 


Detailed Methodology 

The OIG obtained a report from TEAMS II RMIS staff of all the Action Items completed from 
November 7, 2009, to February 7, 2010. The report consisted of 550 Action Items, which were 
all used by the OIG to conduct the following test. 


Test 1: Determine if all Action Items were completed within 60 days of the creation date. 


Testing Results 





The Department was 83% compliant with this objective. Four hundred and fifty-four of the 550 
Action Items tested were completed within 60 days. 


B. Transfer Action Items 


Objective B1: Determine if both the Transfer Action Item and related TEAMS Evaluation 
Report forms were completed in accordance with Special Order No. 23, 2003. 


Background 


When an officer is promoted or transferred to a new assignment, the information is tracked by 
Personnel Division which subsequently publishes a Transfer Order. Once the Transfer Order is 
published, RMIS automatically creates a Transfer Action Item for each affected officer. The 
Transfer Action Item then generates a tracking number for a TEAMS Evaluation Report. A 
TEAMS Evaluation Report form documents the review of an officer’s TEAMS II report and 
specifically requires that the supervisor record such items as sustained administrative 
investigations, adverse judicial findings, and instances of discipline for excessive force, false 
arrest or charge, improper search or seizure, sexual harassment, discrimination, or dishonesty. 


Detailed Methodology 


The OIG obtained a report from TEAMS II RMIS staff of all Transfer Action Items completed 
from September 1, 2009, through December 5, 2009. The OIG then sorted the Transfer Action 
Items by completion date and judgmentally selected the time period of November 8, 2009, 
through December 5, 2009, for this objective. 


The time period selected contained a total of 62 Transfer Action Items. From the 62 Transfer 
Action Items, the OIG selected a random sample of 26 Transfer Action Items by using a 95% 
one-tail confidence level, with a 6% expected error rate, and 6% plus-precision. 


For each of the 26 Transfer Action Items tested, the related TEAMS Evaluation Report was 
requested from the respective commanding officer. 


Training Evaluation and Management System (TEAMS) II Audit 
Page 10 of 19 
1.0 


The following test was used to review the sample of Transfer Action Items: 


Test 1: Determine if the Transfer Action Items and related TEAMS Evaluation Report forms 
were completed appropriately in accordance with Special Order No. 23, 2003. 


Testing Results 





The Department was 100% complaint with this objective. All 26 Transfer Action Items and the 
related TEAMS Evaluation Report forms contained all the required information as set forth in 
Special Order No. 23, 2003. 


Objective B2: Determine if the Transfer Action Items and TEAMS Evaluation Report forms 
were completed within 10 days of publication of the related Transfer Order. 


Background 


According to Special Order No. 23, 2003, entitled “Criteria for Transfer/Loans of Sworn 
Personnel,” when an officer is transferred/loaned, the commanding officer must ensure that the 
officer’s TEAMS II report is reviewed within 10 working days from publication of the Transfer 
Order and/or notification of the loan. 


Detailed Methodology 





The OIG obtained a report from TEAMS II RMIS staff of all Transfer Action Items completed 
from September 1, 2009, through December 5, 2009. The OIG then sorted the Transfer Action 
Items by completion date and judgmentally selected the time period of November 8, 2009, 
through December 5, 2009, for this objective. 


The time period selected contained 62 Transfer Action Items. From the 62 Transfer Action 
Items, the OIG selected a random sample of 26 Transfer Action Items by using a 95% one-tail 


confidence level, with a 6% expected error rate, and 6% plus-precision. 


For each of the 26 Transfer Action Items tested, the related TEAMS Evaluation Report was 
requested from the respective commanding officer. 


The following tests were used to review the sample of Transfer Action Items: 
Test 1: Determine if the Transfer Action Items were completed within 10 days. 
Test 2: Determine if the TEAMS Evaluation Report forms were completed within 10 days. 


Testing Results 


The Department was 92% compliant with this objective. Twenty-four of the 26 Transfer Actions 
Items were completed within the required 10-day deadline. Details for those exceptions are as 
follows: 
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One officer’s Transfer Action Item was completed after the 10-day deadline; however, the 
corresponding TEAMS Evaluation Report form was completed on time." 


One officer’s TEAMS Evaluation Report was not signed by the Commanding Officer within the 
10 day deadline; however, the Transfer Action Item was closed in TEAMS II on time.” 


Objective B3: Determine if Transfer Action Items were generated for all officer transfers. 


Background 


In addition to the previously mentioned information regarding Transfer Action Items, the OIG 
noted that Special Order No. 23, 2003, excludes “probationary police officers who are 
transferring from Training Division.” 


Detailed Methodology 


The OIG obtained a report from TEAMS II RMIS staff of all Transfer Action Items completed 
from September 1, 2009, through December 5, 2009. The Transfer Action Items were then 
sorted according to date. 


The OIG also obtained the Transfer Order for October 11, 2009, through November 7, 2009, 
(Deployment Period 11). The total number of transfers for that time period, excluding 
probationary officers, civilians, reserve officers and the return-to-work program officers, was 90. 
The Transfer Order for Deployment Period 11 was compared to the TEAMS II RMIS staff report 
to conduct the following test: 


Test 1: Determine if Transfer Action Items were generated for all officer transfers. 


Testing Results 





The Department was 100% complaint with this objective. All 90 transfers had the required 
corresponding Transfer Action Items. 


C. Performance Evaluation Report Action Items 
Objective C1: Determine if the Performance Evaluation Report Action Items and related 


Standards Based Assessment forms evaluated the supervisor’s performance in accordance 
with Department Manual Section 3/760.20. 


Background 


A Performance Evaluation Report Action Item notifies a commanding officer that a subordinate 
officer’s annual performance evaluation is due for completion. This type of Action Item ensures 
that the officer’s TEAMS II report is reviewed at the time of the officer’s performance 





18 TAT No. 534519 


1 TAT No. 537808 
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evaluation. Department Manual Section 3/760.20 defines, in part, that a Standards Based 
Assessment form is used to evaluate officers at the rank of lieutenant and below on an annual 
basis. Additionally, Department Manual Section 1/668.01 specifies that for subordinate officer 
evaluations, supervisors are required to review the officer’s Summary of Employee Activity 
report and TEAMS II report. 


Detailed Methodology 


The OIG obtained a report from TEAMS II RMIS staff of all Performance Evaluation Report 
Action Items completed from September 1, 2009, through December 5, 2009. The OIG sorted 
the Performance Evaluation Report Action Items and selected only those items in which 
commanding officers had completed annual performance evaluations for sergeants and 
lieutenants assigned to field duties. The OIG then further sorted the selected Performance 
Evaluation Report Action Items by completion date and judgmentally selected the time period of 
November 8, 2009, through December 5, 2009, for this objective. 


The time period selected contained 38 Performance Evaluation Report Action Items. The OIG 
selected a random sample of 21 Performance Evaluation Report Action Items by using a 95% 
one-tail confidence level, with a 6% expected error rate, and 6% plus-precision. 


For each of the 21 Performance Evaluation Report Action Items tested, the supervisor’s related 
Standards Based Assessment form was requested from the respective commanding officer. In 
addition to the review of the Standards Based Assessment form, the supervisor’s TEAMS II 
report and Summary of Employee Activity report were reviewed. 


The following test was used to review the sample of Performance Evaluation Report Action 
Items: 


Test 1: Determine if the affected commanding officer checked one of the following boxes in 
completing his/her sergeant’s/lieutenant’s annual performance evaluation. 














10 “Regularly reviews various RMIS reports, audits, Action Items using the TEAMS II. 
Completes TEAMS II transfer reviews on time.” 











20 “Seldom or never reviews various RMIS reports, audits, Action Items using the TEAMS II. 
Does not complete the TEAMS II transfer reviews on time.” 





Testing Results 





The Department was 100% complaint with this objective. All 21 Performance Evaluation Report 
Action Items reviewed had the appropriate box checked. 
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Objective C2: Determine if Performance Evaluation Report Action Items were generated for 
related annual performance evaluations. 


Background 


On April 18, 2008, TEAMS II Development Bureau issued an Intradepartmental Correspondence 
outlining the Performance Evaluation Report Action Item process. According to the 
memorandum, TEAMS II automatically generates a Performance Evaluation Report Action Item 
for each officer 30 days prior to their annual anniversary date. 


Detailed Methodology 


The OIG obtained two reports in order to test this objective. First, the OIG obtained a report 
from TEAMS II RMIS staff of all Performance Evaluation Report Action Items created from 
September 1, 2009, through December 5, 2009. 


Second, the OIG obtained a report from TEAMS II RMIS staff of all officers with anniversary 
dates from November 8, 2009, through December 5, 2009. The time period selected contained 
670 related Performance Evaluation Report Action Items. From the 670 Performance Evaluation 
Report Action Items, the OIG selected a random sample of 40 Performance Evaluation Report 
Action Items by using a 95% one-tail confidence level, with a 6% expected error rate, and 6% 
plus-precision. 


The following test was used to review the sample of Performance Evaluation Report Action 
Items: 


Test 1: Determine if there was a Performance Evaluation Report Action Item created for each 
officer in the sample. 


Test Results 


The Department was 100% complaint with this objective. All 40 officers in the sample had a 
Performance Evaluation Report Action Item created. 


Objective C3: Determine if Performance Evaluation Report Action Items were completed 
within 60 days. 


Background 


On April 18, 2008, TEAMS II Development Bureau issued an Intradepartmental Correspondence 
outlining the Performance Evaluation Report Action Item process. According to the 
memorandum, an officer’s Performance Evaluation Report Action Item and the related Standards 
Based Assessment form are required to be completed within 60 days following the officer’s 
anniversary date. 
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Detailed Methodology 


The OIG obtained a report from TEAMS II RMIS staff of all Performance Evaluation Report 
Action Items completed from September 1, 2009, through December 5, 2009. The total 
population consisted of 2,201 Performance Evaluation Report Action Items. The OIG used the 
total population to conduct the following test. 


Test 1: Determine if all Performance Evaluation Report Action Items were completed within 60 
days following an officer’s anniversary date. 


Testing Results 


The Department was 69% compliant with this objective. One thousand five hundred and twenty- 
six of the 2,201 performance Action Items tested were completed within 60 days. The 675 
exceptions ranged from one to 109 days past due. All of the exceptions are based on the Action 
Item closure date and not the signed date of the related Standards Based Assessment forms. 


D. Supervisor Action Items 


Objective D1: Determine if Supervisor Action Item dispositions were adequately reviewed and 
justified in accordance with Special Order No. 28, 2008. 


Background 


Supervisor Action Items allow supervisors and commanding officers to monitor a subordinate’s 
work performance. A Supervisor Action Item is manually generated by an officer’s supervisor 
using RMIS. The information required to complete each Supervisor Action Item is contained on 
the affected officer’s TEAMS II report. 


Special Order No. 28, 2008, states, in relevant part, when generating a Supervisor Action Item a 
supervisor “should document the extent of their review and justify any corrective action 


necessary.” 


Detailed Methodology 


The OIG obtained a report from TEAMS II RMIS staff of all Supervisor Action Items completed 
from September 1, 2009, through December 5, 2009. The OIG then sorted the Supervisor Action 
Items by the completion date and judgmentally selected the time period of November 8, 2009, 
through December 5, 2009, for this objective. 





°° Corrective actions a supervisor may choose include: cancel, no action, commendation, informal meeting, training, 
special evaluation reports, modified field duties, assigned to non-field duties, Risk Management Executive 
Committee referral, directed Behavioral Science Services referral, comment card, notice to correct deficiencies, and 
complaint. 
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The time period selected contained 74 Supervisor Action Items. From the 74 Supervisor Action 
Items, the OIG selected a random sample of 28 Supervisor Action Items by using a 95% one-tail 
confidence level, with a 6% expected error rate, and 6% plus-precision. 


Each Supervisor Action Item was examined to determine if the supervisor and reviewers had 
adequately documented their investigation in accordance with Special Order No. 28, 2008. 
Further, each Supervisor Action Item was compared to the officer’s TEAMS II report and 
Summary of Employee Activity report in order to verify the narrative and comments. 


The following tests were used to review the sample of Supervisor Action Items: 


Test 1: Determine if there was a reason stated for the creation of the Supervisor Action Item in 
the narrative. 


Test 2: Determine if adequate justification was provided for the selected disposition in 
accordance with Special Order No. 28, 2008. 


Test 3: Determine if the TEAMS II report was adequately reviewed for each Supervisor Action 
Item. 


Test Results 


The Department was 96% compliant with this objective. Twenty-seven of the 28 Supervisor 
Action Items tested contained the required aforementioned information. The exception was 
found as a result of Test 3; the Supervisor Action Item narrative did not consider information 
which was available in the affected officer’s TEAMS II report. Details for the one exception are 
as follows: 


In one Supervisor Action Item, the officer involved received a disposition of “comment card” as 
a corrective action in respect to a personnel complaint.” The officer’s TEAMS II report 
however, reflected two prior personnel complaints for similar types of actions.” According to 
the commanding officer as it was related to the OIG, the affected officer’s TEAMS II report had 
not been adequately reviewed for this Supervisor Action Item. Had adequate review of the 
Officer’s TEAMS II report been conducted, it would have led to a different corrective action. 


Objective D2: Determine if there was evidence that corrective action was taken for Supervisor 
Action Items with corrective action dispositions. 


Background 


Following a supervisor’s review of the officer’s TEAMS II report, the supervisor may determine 
that a corrective action is warranted. The corrective actions available to the supervisor include: 
training, special evaluation reports, modified field duties, assigned to non-field duties, risk 
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management executive committee, directed Behavioral Science Services referral, comment card, 
notice to correct deficiencies, and complaint. 
Detailed Methodology 





The OIG obtained a report from TEAMS II RMIS staff of all Supervisor Action Items completed 
from September 1, 2009, through December 5, 2009. The OIG then sorted the listing by the 
completion date and judgmentally selected the time period of November 8, 2009, through 
December 5, 2009, for this objective. 

The time period selected contained 74 Supervisor Action Items. From the 74 Supervisor Action 
Items, the OIG selected a random sample of 28 Supervisor Action Items by using a 95% one-tail 
confidence level, with a 6% expected error rate, and 6% plus-precision. The sample of 28 
Supervisor Action Items was then sorted by corrective action dispositions. Ten of the 28 
corrective action dispositions were comment card. To test for this objective, the OIG requested 
the related comment cards from the respective commanding officers. The OIG used the total 
population of 10 comment cards to test the Supervisor Action Items. 


The following test was used to review the Supervisor Action Items: 


Test 1: Determine if there was evidence that corrective actions were taken for the selected 
Supervisor Action Items. 


Test Results 


The Department was 90% compliant with this objective. Nine of the 10 Supervisor Action Items 
tested contained documentation indicating the corrective action had been taken. The exception 
was a result of one area command being unable to locate the requested comment card. Details 
for the one exception are as follows: 


One Supervisor Action Item did not have proof of the corrective action due to the affected 
officer’s comment card not being located.” 


RECOMMENDATIONS 


The supervisor’s analysis of an Action Item, according to Special Order No. 28, 2008, requires 
justification of the corrective action disposition and that the corrective action taken is a direct 
result of the Action Item review. Corrective action imposed by a supervisor and taken prior to 
the creation of the Action Item, and which requires no additional action, is properly closed with 
the disposition of “no action.” The objective testing this requirement, A2, revealed a compliance 
rate of 39%. Although this objective is procedural and without impact on the supervisor’s 
ultimate determination of an officer’s potential to engage in at-risk behavior, adherence to the 
protocol is valuable in ensuring the effectiveness of the TEAMS II system. Based on the 
compliance rate, it appears that the requirements of this objective are broadly misunderstood. As 
such, the OIG recommends that the Department provide department-wide training relating to the 
relevant protocols established by Special Order No. 28, 2008. 





3 SAT No. 537794 


Training Evaluation and Management System (TEAMS) II Audit 
Page 17 of 19 
1.0 


Special Order No. 28, 2008, as well as the Transition Agreement, requires that a supervisor, in 
response to an Action Item, analyze all relevant TEAMS II information to detect any pattern or 
series of incidents that indicate that an officer may be engaging in at-risk behavior. Neither 
document, however, provides guidance to assist the supervisor in making the determination as to 
whether an officer’s behavior denotes a “pattern” or should be considered “at-risk.” Absent 
Department guidance as to the meaning of these terms, it is unrealistic to expect supervisors to 
administer a consistent interpretation of the behavior in question. Quality and consistency of 
supervisory oversight is promoted through the understanding and adherence to defined standards. 
As such, to provide a consistent quality of supervisory oversight, the OIG recommends that the 
Department develop guidance for supervisors to use in determining “patterns” and “at-risk 
behavior.” Additionally, the OIG recommends that the Department design a system to 
implement the use of the guidance in the analysis of Action Items and, subsequently, that the 
Department audit compliance with the appropriate application of this system. 


MANAGEMENT RESPONSE 
Information Technology Bureau’s Audit response has been attached as an addendum. 
CONCLUSION 


The OIG has completed the TEAMS II Audit which reviewed Action Items, Transfer Action 
Items, Performance Evaluation Report Action Items and Supervisor Action Items. In general, it 
appears that supervisors are utilizing the TEAMS II system to assess potential patterns and 
practices of behavior that may lead to officer misconduct. The OIG noted that compliance rates 
for most Audit objectives are significantly higher than in past audits. 
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ADDENDUM 


TEAMS II DIVISION RESPONSE TO THE OFFICE OF THE INSPECTION GENERAL’S 
AUDIT FOR FISCAL YEAR 2009/2010 


69% Compliance Rate on PER Completion within 60 Days (page 14) 


This audit determined compliance based on the electronic PER completion date, rather 
than the date the evaluated employee signed the hardcopy form, which has been the basis 
used in previous audits. In the past, auditors have acknowledged that the hardcopy 
should be given more weight because there is a lag between the time the hardcopy is 
signed by the employee and the time the electronic PER is closed out in the system. 


Recommendation #1 (page 16) 


TEAMS II Division will provide update training to all Bureau TEAMS II Coordinators 
on the proper use of Action Item dispositions. The training will provide guidance on the 
use of the various AI dispositions and remind users that dispositions selected are actions 
that will be taken as a result of the AI review. This training will also be included in 
TEAMS II Coordinator meetings, Command Staff Development training and supervisor 
school. 


TEAMS II Division will also be sending out a notice department-wide, delineating the 
information provided in this training. 


Improvements in compliance rates from the last audit are as follows: 


The OIG audit conducted in July 2009 focused on TAIs and PERs. Numbers in bold 
shows marked improvement from the last audit. 
































2009 2010 
a) TAI and TER completed appropriately 100% 100% 
b) TAI and TER completed within 10 days in 92% 
c) TAIs generated for all transfers 100% 100% 
d) PERAI and SBA evaluated the supervisor’s 61% 100% 

performance 

e) PERAI generated for all annual evaluations due 100% 100% 
f) PERAI completed within 60 days 45% 69% 








* methodology used in 2009 was different therefore compliance rates are not 
comparable 
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Several improvements have been made to the system, and a variety of update training 
continues to be provided to users. 


1. The Investigative Narrative guidelines implemented in February 2009 added eight 
preset questions to assist supervisors in responding to Action Items. This helped in 
providing for a more comprehensive and quality review of officer performance. 


2. The creation of the Transfer Action Item Form, which replaced the TEAMS 
Evaluation Report, eliminated confusion experienced by users with the old form and 
facilitated the completion of the TAI. 


3. The incorporation of the Transfer Action Item form into the electronic TAI provided 
for a more seamless and efficient process for the completion of TAIs. 


4. TEAMS II Division continues to conduct quarterly TEAMS II Coordinator meetings 
to provide continuous training and obtain feedback from users. Implementation of 
minor enhancements to the system from these meetings has brought about 
improvements in response times. 


5. TEAMS II Division continues to conduct regular classroom training for new TEAMS 
II Coordinators, Supervisor School, Command Staff Development and TEAMS II 
Refresher Courses. 


